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DTHAMIC  PROORAJUCDft} ,  HCNLIXEAR  VARIATIONAL 
PROCESSES ,  AMD  SUCCESSIVE  APPROXIMATIONS 

Richard  Bellman 

1.  DITRODUCTIOM 

(Xir  object  in  this  paper  is  to  show  that  a  blend  of  dynamic 
programming,  eucoeseive  approximations  Mid  digital  computer# 
enables  us  to  approach  various  classes  of  nonlinear  variational 
problems  formerly  far  beyond  our  reach. 

The  general  problem  we  with  to  consider  is  one  which 
arises  in  many  parts  of  analysis,  mathematical  physics  and 
applied  mathematics,  namely  that  of  determining  the  minimum  of 
a  functional  of  the  form 

J(v)  -  /  T  , .  . .  ,x^(t))dt  ♦  0(x1(T) , . . .  ,Xy(T) ) ,  (1.1) 

>.ver  all  forcing  functions  v^(t)  subject  to  relations  of  the 
type 

<3*4 

®  ^  ^  ^  ^  ( I ) »  ^^  ( 0 )  m  ®  i  *  ^  *  1#2,...,N 

b*  "0  KJ^V1»V2'* *  * ,VN^dt  ^  bJ'  J  -  1,2,.. ,,L  (1.2) 

c.  p1(x1,...#x1(jt)  ^  vA(t)  ^  q1(x1#...,xNit)#  0  £  t  £  T. 

1  2 

As  we  have  ahown  elsewhere  '  ,  the  computational  eolutlon 
of  questions  of  this  type  can  be  transformed  by  means  of  the 
functional  equation  technique  of  dynamic  programming  Alto  problems 
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involving  the  determination  of  sequences  of  function*  of  M 
variables  using  elamen tAry  operation*.  If#  however,  N  £  5, 
the  limited  memory  and  speed  of  current  digital  computer* 
prevents  this  fro®  being  a  routine  technique  of  solution  at 
the  present  time. 

It  follows  that  if  we  wish  to  attack  general  problems 
of  this  nature,  we  must  introduce  more  refined  techniques. 

2.  THE  LINEAR  CASK 

The  starting  point  of  our  investigation  is  the  result 
contained  in  an  earlier  paper  on  control  processes  ^  in  which 
it  is  shown  that  in  the  case  where  the  underlying  equations  of 
(l.2a)  are  linear,  the  criterion  functions  t  and  0  are 
linear,  and  the  constraints  of  (1.2c)  have  the  simpler  form 

Pi  ( t )  £  v^t)  <  Q1(t),  0  £  t  £  T,  (2.1) 

then  che  variational  problem  described  in  the  preceding  section 
can  be  reduced  to  the  computation  of  a  sequence  of  functions  of 
one  variable,  regardless  of  the  size  of  N.  If  the  quantity  L 
appearing  In  ( 1 . 2b )  Is  equal  to  one,  we  can  do  this  directly;  If 
It  Is  greater  than  one,  we  employ  Lagrange  r.ultipliers 

It  was  also  shown  that  if  the  functions  f  and  H  are 
linear,  and  0  Is  a  runctlon  of  only  k  of  the  N  components 
of  x(T),  then  a  computational  solution  can  be  effected  in  terms 
of  functions  of  k  variables. 

Tills  fact  Is  important  in  connection  with  engineering  control 
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prootiNs  involving  time— lags  and  eoonoeilc  control  proceesea 
involving  *  ccmc  lex  of  Industries . 

These  results  combined  with  the  claaaical  tool  of 
sucoesslve  approximations  will  permit  us  to  treat  various 
pert  a  of  the  gene  re  1  p'.x>blem. 

5.  guccsaaivi  /t?pmxpuTiowa 

Let  ua  ncrw  outline  how  the  results  obtained  for  the  linear 
case,  aided  and  abetted  by  successive  appreciations ,  provide 
a  feasible  achense  for  computing  tha  solutions  of  nonlinear 
problems^  in  terme  of  sequences  of  functions  of  one  variable. 

We  shall  consider  first  the  case  where  we  are  interested 
in  terminal  control  with  Q  linear  In  it*  argxments.  Let 
v^(t),  1  •  1,  2,  .  ..,  K,  be  an  Initial  choice  of  forcing 

function  satisfying  (1.2a)  and  (l.2b)  and  let  the  functione 

0/  X 

x^  (t)  be  determined  by  neons  of  the  equation* 

dx.  0 

3T“  -  H1(x1#...#i1|)  ♦  vt  (t ),  x1(0)  -  o1# 

(3 

1  -  1,  2,  ...,  W. 

To  obtain  the  next  approximation,  v^(t),  consider  the 
problem  of  minimizing  0( x^ (T) ,x^(  T) , . . . , x^( T) )  over  all 
functions  v^(t)  SAtlafying  ( 1 . 2a  )  and  ( 1 . 2t )  where  the 
function!  x^(t)  are  now  deterr.lned  by  the  approximating 


linear  relation* 
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Ax,  A  -  N  n  o>H- 

ST"  •  ♦^(xj-Xj)  —g  + 


♦  V. (t). 


*,(0) 


sl* 


(3. 


Since  thee#  equation*  are  linear,  and  <1  ha*  been  uraed  to 
be  linear,  the  minimising  sequence,  tJ,  can  now  be  computed 
via  sequences  of  function*  of  on*  variable.  In  soa*  oaae*,the 
solution  can  be  obtained  analytically,  cf.  thus  greatly 

simplifying  the  application  of  this  method. 

Once  the  new  forcing  functions,  v^(t),  have  been 
determined,  the  new  state  functions,  x^(t),  are  detemlned 
by  way  of  (3.1),  with  v®  replaced  by  v^. 

Repeating  this  process,  step-by-step,  w*  dete mine  a 
sequence  of  forcing  functions,  |*i(*)j#  *  sequence  of 

state  functions,  jx£(t)j  . 

».  HOHOTOMICITY  OF  APPROX DUTIOW 

Without  prejudicing  ourselves  In  the  matter  of  convergence. 
It  Is  easy  to  show  that  we  have  monotonoclty  of  approximation 
In  the  sense  that 

0(x®(T),...,x®(T))  £  0(xJ(T),...,xJ(T))  £  ... 

(%. 

>  0( (T) , . . • ,Xg(T) )  ^  ••• 

To  see  this,  observe  that  If  v^(t)  Is  taken  equal  to  vj(t) 

In  (3*2),  we  obtain  a  system  of  llnoar  equations  whose  solution 
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la  clearly  x^  •  x®.  It  follow*  that  a  aat  of  forcing  functions 
Vj  which  minimise  must  yl*ld  a  wlue  of  0(x1(T) ,  . . . #*W(T) ) 
which  la  at  most  0(x®(T) , . . .  ,xjj(T) ) .  The  general  result 
follow*  lnduotlvely. 

This  monotonlolty  of  approximation  la  not  surprising, 

considering  that  w*  are  esaentlally  carrying  out  an  approximation 
7 

In  policy  space  . 

5.  WOlfLIIgAR  CRH1RI0M  FUNCTION — I 

To  begin  with,  we  note  that  the  general  problem  described 
In  (l.l)  can  always  be  reduced  to  a  terminal  control  process  by 
the  Introduction  of  a  new  dependent  variable  x^4l  defined  by 
the  differential  relation 


*(*1#*2#  •  •  •  •  *jj^i(®) 


0. 


(5.1) 


If  0  Is  nonlinear,  but  contains  only  k  of  the  N  components 
of  x(T),  then,  as  Indicated  above,  the  approximation  procedure 
reduces  the  computational  problem  to  one  Involving  sequences  of 
functions  of  k  variables. 

6.  KQHLDfiAfi  carnation  function — ii 

Alternatively,  we  can  approximate  to  0  not  only  by  means 
of  linear  terms,  but  more  accurately  by  means  of  a  second  degree 
expression.  Having  done  this,  we  can  take  advantage  of  the  fact 
that  the  minimization  of  a  quadratic  functional  of  the  form 
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J(v)  •  /'T  [(x,B(t)x)  ♦  (x,b(t) )  ♦  (v,C(t)v) 
w0 

(6.1) 

♦  (v,c(t)J]dt  ♦  (x(T) ,Qx(T) )  ♦  (x(T),g) 
where  x  and  v  art  connected  by  means  of  a  linear  equation 

-  A(t)x  ♦  v,  x(0)  -  c ,  (6.2) 

with  no  other  constraints,  depends  upon  the  solution  of  linear 
equations;  cf.  ^  where  classical  techniques  are  esqployed. 

Dynamic  programing  techniques,  utilizing  the  fact  that  the 
minimum  Is  a  quadratic  function  of  the  c^,  the  oooqxuients  of  c, 
can  also  be  applied. 

This  affords  a  ntw  approach  to  bosm  general  claases  of 
nonlinear  equations  and  nonlinear  variational  problems  which 
we  will  discuss  In  greater  detail  subsequently. 

7.  DI8CP88I0H 

There  are  a  number  of  other  types  of  approximations  which 
we  cannot  present  here  due  to  llsdtatlons  of  space.  There  are 
also  a  large  number  of  Interesting  analytic  questions  concerning 
convergence,  local  minima,  rapldlt?  of  convergence,  etc.,  to 
be  discussed  at  a  later  date  in  connection  with  the  foregoing 
techniques. 
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